Method and apparatus for sub-block based temporal motion vector prediction

ABSTRACT

A method of video decoding includes acquiring a current and identifying, for a current block included in the current picture, a reference block included in a reference picture that is different from the current picture, where the current block is divided into a plurality of sub-blocks (CBSBs), and the reference block has a plurality of sub-blocks (RBSBs). The method includes, determining whether the reference picture for the RBSB is the current picture, and in response to determining that the reference picture for the RBSB is the current picture, determining a coding mode of the RBSB as an intra mode. The method further includes, in response to determining that the reference picture for the RBSB is not the current picture determining a motion vector predictor for the one of the CBSBs based on whether the coding mode of the corresponding RBSB is one of the intra mode and the inter mode.

INCORPORATION BY REFERENCE

This application is a continuation of and claims priority to U.S. patentapplication Ser. No. 16/210,635, filed Dec. 5, 2018, which claims thebenefit of priority to U.S. Provisional Application No. 62/680,468,“METHODS FOR SUB-BLOCK BASED TEMPORAL MOTION VECTOR PREDICTION” filed onJun. 4, 2018. The benefit of priority is claimed to each of theforegoing, and the entire contents of each of the foregoing areincorporated herein by reference.

TECHNICAL FIELD

The present disclosure describes embodiments generally related to videocoding.

BACKGROUND

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent the work is described in thisbackground section, as well as aspects of the description that may nototherwise qualify as prior art at the time of filing, are neitherexpressly nor impliedly admitted as prior art against the presentdisclosure.

Video coding and decoding using inter-picture prediction with motioncompensation has been known for decades. Uncompressed digital video caninclude a series of pictures, each picture having a spatial dimensionof, for example, 1920×1080 luminance samples and associated chrominancesamples. The series of pictures can have a fixed or variable picturerate (informally also known as frame rate), of, for example 60 picturesper second or 60 Hz. Uncompressed video has significant bitraterequirements. For example, 1080p60 4:2:0 video at 8 bit per sample(1920×1080 luminance sample resolution at 60 Hz frame rate) requiresclose to 1.5 Gbit/s bandwidth. An hour of such video requires more than600 GByte of storage space.

One purpose of video coding and decoding can be the reduction ofredundancy in the input video signal, through compression. Compressioncan help reduce aforementioned bandwidth or storage space requirements,in some cases by two orders of magnitude or more. Both lossless andlossy compression, as well as a combination thereof can be employed.Lossless compression refers to techniques where an exact copy of theoriginal signal can be reconstructed from the compressed originalsignal. When using lossy compression, the reconstructed signal may notbe identical to the original signal, but the distortion between theoriginal and reconstructed signal is small enough to make thereconstructed signal useful for the intended application. In the case ofvideo, lossy compression is widely employed. The amount of distortiontolerated depends on the application; for example, users of certainconsumer streaming applications may tolerate higher distortion thanusers of television contribution applications. The compression ratioachievable can reflect that: higher allowable/tolerable distortion canyield higher compression ratios.

Motion compensation can be a lossy compression technique and can relateto techniques where a block of sample data from a previouslyreconstructed picture or part thereof (reference picture), after beingspatially shifted in a direction indicated by a motion vector (MVhenceforth), is used for the prediction of a newly reconstructed pictureor picture part. In some cases, the reference picture can be the same asthe picture currently under reconstruction. MVs can have two dimensionsX and Y, or three dimensions, the third being an indication of thereference picture in use (the latter, indirectly, can be a timedimension).

In some video compression techniques, an MV applicable to a certain areaof sample data can be predicted from other MVs, for example from thoserelated to another area of sample data spatially adjacent to the areaunder reconstruction, and preceding that MV in decoding order. Doing socan substantially reduce the amount of data required for coding the MV,thereby removing redundancy and increasing compression. MV predictioncan work effectively, for example, because when coding an input videosignal derived from a camera (known as natural video) there is astatistical likelihood that areas larger than the area to which a singleMV is applicable move in a similar direction and, therefore, can in somecases be predicted using a similar motion vector derived fromneighboring area's MVs. That results in the MV found for a given area tobe similar or the same as the MV predicted from the surrounding MVs, andthat in turn can be represented, after entropy coding, in a smallernumber of bits than what would be used if coding the MV directly. Insome cases, MV prediction can be an example of lossless compression of asignal (namely: the MVs) derived from the original signal (namely: thesample stream). In other cases, MV prediction itself can be lossy, forexample because of rounding errors when calculating a predictor fromseveral surrounding MVs.

Various MV prediction mechanisms are described in H.265/HEVC (ITU-T Rec.H.265, “High Efficiency Video Coding”, December 2016). Out of the manyMV prediction mechanisms that H.265 offers, described here is atechnique henceforth referred to as “spatial merge”.

Some forms of inter-prediction are performed on a sub-block level.However sub-block based temporal motion vector prediction modes, such asalternative temporal motion vector prediction (ATMVP) andspatial-temporal motion vector prediction (STMVP), require that acorresponding sub-block be coded in inter mode. However, these temporalmotion vector prediction modes are unable to handle sub-blocks that arecoded in an intra mode such as intra block copy mode.

SUMMARY

An exemplary embodiment of the present disclosure includes a method ofvideo decoding for a decoder. The method includes acquiring a currentpicture from a coded video bitstream. The method further includesidentifying, for a current block included in the current picture, areference block included in a reference picture that is different fromthe current picture, where the current block is divided into a pluralityof sub-blocks (CBSBs), and the reference block has a plurality ofsub-blocks (RBSBs) that each correspond to a different one of theplurality of CBSBs. The method further includes determining whether thereference picture for the RBSB is the current picture, and in responseto determining that the reference picture for the RBSB is the currentpicture, determining a coding mode of the RBSB as an intra mode. Themethod further includes, in response to determining that the referencepicture for the RBSB is not the current picture, (i) determining, forone of the CBSBs, whether a coding mode of the RBSB is one of an intramode and an inter mode, and (ii) determining a motion vector predictorfor the one of the CBSBs based on whether the coding mode of thecorresponding RBSB is one of the intra mode and the inter mode.

An exemplary embodiment of the present disclosure includes a videodecoder for video decoding. The video decoder includes processingcircuitry configured to acquire a current picture from a coded videobitstream. The processing circuitry is further configured to identify,for a current block included in the current picture, a reference blockincluded in a reference picture that is different from the currentpicture, where the current block is divided into a plurality ofsub-blocks (CBSBs), and the reference block has a plurality ofsub-blocks (RBSBs) that each correspond to a different one of theplurality of CBSBs. The processing circuitry is further configured todetermine whether the reference picture for the RBSB is the currentpicture, and in response to the determination that the reference picturefor the RBSB is the current picture, determine a coding mode of the RBSBas an intra mode. The processing circuitry is further configured to, inresponse to the determination that the reference picture for the RBSB isnot the current picture, (i) determine, for one of the CBSBs, whether acoding mode of the RBSB is one of an intra mode and an inter mode, and(ii) determine a motion vector predictor for the one of the CBSBs basedon whether the coding mode of the corresponding RBSB is one of the intramode and the inter mode.

An exemplary embodiment of the present disclosure includes anon-transitory computer readable medium having instructions storedtherein, which when executed by a processor in a video decoder causesthe processor to execute a method. The method includes acquiring acurrent picture from a coded video bitstream. The method furtherincludes identifying, for a current block included in the currentpicture, a reference block included in a reference picture that isdifferent from the current picture, where the current block is dividedinto a plurality of sub-blocks (CBSBs), and the reference block has aplurality of sub-blocks (RBSBs) that each correspond to a different oneof the plurality of CBSBs. The method further includes determiningwhether the reference picture for the RBSB is the current picture, andin response to determining that the reference picture for the RBSB isthe current picture, determining a coding mode of the RBSB as an intramode. The method further includes, in response to determining that thereference picture for the RBSB is not the current picture, (i)determining, for one of the CBSBs, whether a coding mode of the RBSB isone of an intra mode and an inter mode, and (ii) determining a motionvector predictor for the one of the CBSBs based on whether the codingmode of the corresponding RBSB is one of the intra mode and the intermode.

BRIEF DESCRIPTION OF THE DRAWINGS

Further features, the nature, and various advantages of the disclosedsubject matter will be more apparent from the following detaileddescription and the accompanying drawings in which:

FIG. 1 is a schematic illustration of a simplified block diagram of acommunication system (100) in accordance with an embodiment.

FIG. 2 is a schematic illustration of a simplified block diagram of acommunication system (200) in accordance with an embodiment.

FIG. 3 is a schematic illustration of a simplified block diagram of adecoder in accordance with an embodiment.

FIG. 4 is a schematic illustration of a simplified block diagram of anencoder in accordance with an embodiment.

FIG. 5 shows a block diagram of an encoder in accordance with anotherembodiment.

FIG. 6 shows a block diagram of a decoder in accordance with anotherembodiment.

FIG. 7 is a schematic illustration of intra picture block compensation.

FIG. 8 is a schematic illustration of a current block and surroundingspatial merge candidates of the current block.

FIG. 9 is a schematic illustration of sub-blocks of a current block andcorresponding sub-blocks of a reference block.

FIG. 10 illustrates an embodiment of a process performed by an encoderor decoder.

FIG. 11 is a schematic illustration of a computer system in accordancewith an embodiment.

DETAILED DESCRIPTION OF EMBODIMENTS

FIG. 1 illustrates a simplified block diagram of a communication system(100) according to an embodiment of the present disclosure. Thecommunication system (100) includes a plurality of terminal devices thatcan communicate with each other, via, for example, a network (150). Forexample, the communication system (100) includes a first pair ofterminal devices (110) and (120) interconnected via the network (150).In the FIG. 1 example, the first pair of terminal devices (110) and(120) performs unidirectional transmission of data. For example, theterminal device (110) may code video data (e.g., a stream of videopictures that are captured by the terminal device (110)) fortransmission to the other terminal device (120) via the network (150).The encoded video data can be transmitted in the form of one or morecoded video bitstreams. The terminal device (120) may receive the codedvideo data from the network (150), decode the coded video data torecover the video pictures and display video pictures according to therecovered video data. Unidirectional data transmission may be common inmedia serving applications and the like.

In another example, the communication system (100) includes a secondpair of terminal devices (130) and (140) that performs bidirectionaltransmission of coded video data that may occur, for example, duringvideoconferencing. For bidirectional transmission of data, in anexample, each terminal device of the terminal devices (130) and (140)may code video data (e.g., a stream of video pictures that are capturedby the terminal device) for transmission to the other terminal device ofthe terminal devices (130) and (140) via the network (150). Eachterminal device of the terminal devices (130) and (140) also may receivethe coded video data transmitted by the other terminal device of theterminal devices (130) and (140), and may decode the coded video data torecover the video pictures and may display video pictures at anaccessible display device according to the recovered video data.

In the FIG. 1 example, the terminal devices (110), (120), (130) and(140) may be illustrated as servers, personal computers and smart phonesbut the principles of the present disclosure may be not so limited.Embodiments of the present disclosure find application with laptopcomputers, tablet computers, media players and/or dedicated videoconferencing equipment. The network (150) represents any number ofnetworks that convey coded video data among the terminal devices (110),(120), (130) and (140), including for example wireline (wired) and/orwireless communication networks. The communication network (150) mayexchange data in circuit-switched and/or packet-switched channels.Representative networks include telecommunications networks, local areanetworks, wide area networks and/or the Internet. For the purposes ofthe present discussion, the architecture and topology of the network(150) may be immaterial to the operation of the present disclosureunless explained herein below.

FIG. 2 illustrates, as an example for an application for the disclosedsubject matter, the placement of a video encoder and a video decoder ina streaming environment. The disclosed subject matter can be equallyapplicable to other video enabled applications, including, for example,video conferencing, digital TV, storing of compressed video on digitalmedia including CD, DVD, memory stick and the like, and so on.

A streaming system may include a capture subsystem (213), that caninclude a video source (201), for example a digital camera, creating forexample a stream of video pictures (202) that are uncompressed. In anexample, the stream of video pictures (202) includes samples that aretaken by the digital camera. The stream of video pictures (202),depicted as a bold line to emphasize a high data volume when compared toencoded video data (204) (or coded video bitstreams), can be processedby an electronic device (220) that includes a video encoder (203)coupled to the video source (201). The video encoder (203) can includehardware, software, or a combination thereof to enable or implementaspects of the disclosed subject matter as described in more detailbelow. The encoded video data (204) (or encoded video bitstream (204)),depicted as a thin line to emphasize the lower data volume when comparedto the stream of video pictures (202), can be stored on a streamingserver (205) for future use. One or more streaming client subsystems,such as client subsystems (206) and (208) in FIG. 2 can access thestreaming server (205) to retrieve copies (207) and (209) of the encodedvideo data (204). A client subsystem (206) can include a video decoder(210), for example, in an electronic device (230). The video decoder(210) decodes the incoming copy (207) of the encoded video data andcreates an outgoing stream of video pictures (211) that can be renderedon a display (212) (e.g., display screen) or other rendering device (notdepicted). In some streaming systems, the encoded video data (204),(207), and (209) (e.g., video bitstreams) can be encoded according tocertain video coding/compression standards. Examples of those standardsinclude ITU-T Recommendation H.265. In an example, a video codingstandard under development is informally known as Versatile Video Codingor VVC. The disclosed subject matter may be used in the context of VVC.

It is noted that the electronic devices (220) and (230) can includeother components (not shown). For example, the electronic device (220)can include a video decoder (not shown) and the electronic device (230)can include a video encoder (not shown) as well.

FIG. 3 shows a block diagram of a video decoder (310) according to anembodiment of the present disclosure. The video decoder (310) can beincluded in an electronic device (330). The electronic device (330) caninclude a receiver (331) (e.g., receiving circuitry). The video decoder(310) can be used in the place of the video decoder (210) in the FIG. 2example.

The receiver (331) may receive one or more coded video sequences to bedecoded by the video decoder (310); in the same or another embodiment,one coded video sequence at a time, where the decoding of each codedvideo sequence is independent from other coded video sequences. Thecoded video sequence may be received from a channel (301), which may bea hardware/software link to a storage device which stores the encodedvideo data. The receiver (331) may receive the encoded video data withother data, for example, coded audio data and/or ancillary data streams,that may be forwarded to their respective using entities (not depicted).The receiver (331) may separate the coded video sequence from the otherdata. To combat network jitter, a buffer memory (315) may be coupled inbetween the receiver (331) and an entropy decoder/parser (320) (“parser(320)” henceforth). In certain applications, the buffer memory (315) ispart of the video decoder (310). In others, it can be outside of thevideo decoder (310) (not depicted). In still others, there can be abuffer memory (not depicted) outside of the video decoder (310), forexample to combat network jitter, and in addition another buffer memory(315) inside the video decoder (310), for example to handle playouttiming. When the receiver (331) is receiving data from a store/forwarddevice of sufficient bandwidth and controllability, or from anisosynchronous network, the buffer memory (315) may not be needed, orcan be small. For use on best effort packet networks such as theInternet, the buffer memory (315) may be required, can be comparativelylarge and can be advantageously of adaptive size, and may at leastpartially be implemented in an operating system or similar elements (notdepicted) outside of the video decoder (310).

The video decoder (310) may include the parser (320) to reconstructsymbols (321) from the coded video sequence. Categories of those symbolsinclude information used to manage operation of the video decoder (310),and potentially information to control a rendering device such as arender device (312) (e.g., a display screen) that is not an integralpart of the electronic device (330) but can be coupled to the electronicdevice (330), as was shown in FIG. 3. The control information for therendering device(s) may be in the form of Supplementary EnhancementInformation (SEI messages) or Video Usability Information (VUI)parameter set fragments (not depicted). The parser (320) mayparse/entropy-decode the coded video sequence that is received. Thecoding of the coded video sequence can be in accordance with a videocoding technology or standard, and can follow various principles,including variable length coding, Huffman coding, arithmetic coding withor without context sensitivity, and so forth. The parser (320) mayextract from the coded video sequence, a set of subgroup parameters forat least one of the subgroups of pixels in the video decoder, based uponat least one parameter corresponding to the group. Subgroups can includeGroups of Pictures (GOPs), pictures, tiles, slices, macroblocks, CodingUnits (CUs), blocks, Transform Units (TUs), Prediction Units (PUs) andso forth. The parser (320) may also extract from the coded videosequence information such as transform coefficients, quantizer parametervalues, motion vectors, and so forth.

The parser (320) may perform entropy decoding/parsing operation on thevideo sequence received from the buffer memory (315), so as to createsymbols (321).

Reconstruction of the symbols (321) can involve multiple different unitsdepending on the type of the coded video picture or parts thereof (suchas: inter and intra picture, inter and intra block), and other factors.Which units are involved, and how, can be controlled by the subgroupcontrol information that was parsed from the coded video sequence by theparser (320). The flow of such subgroup control information between theparser (320) and the multiple units below is not depicted for clarity.

Beyond the functional blocks already mentioned, the video decoder (310)can be conceptually subdivided into a number of functional units asdescribed below. In a practical implementation operating undercommercial constraints, many of these units interact closely with eachother and can, at least partly, be integrated into each other. However,for the purpose of describing the disclosed subject matter, theconceptual subdivision into the functional units below is appropriate.

A first unit is the scaler/inverse transform unit (351). Thescaler/inverse transform unit (351) receives a quantized transformcoefficient as well as control information, including which transform touse, block size, quantization factor, quantization scaling matrices,etc. as symbol(s) (321) from the parser (320). The scaler/inversetransform unit (351) can output blocks comprising sample values, thatcan be input into aggregator (355).

In some cases, the output samples of the scaler/inverse transform (351)can pertain to an intra coded block; that is: a block that is not usingpredictive information from previously reconstructed pictures, but canuse predictive information from previously reconstructed parts of thecurrent picture. Such predictive information can be provided by an intrapicture prediction unit (352). In some cases, the intra pictureprediction unit (352) generates a block of the same size and shape ofthe block under reconstruction, using surrounding already reconstructedinformation fetched from the current picture buffer (358). The currentpicture buffer (358) buffers, for example, partly reconstructed currentpicture and/or fully reconstructed current picture. The aggregator(355), in some cases, adds, on a per sample basis, the predictioninformation the intra prediction unit (352) has generated to the outputsample information as provided by the scaler/inverse transform unit(351).

In other cases, the output samples of the scaler/inverse transform unit(351) can pertain to an inter coded, and potentially motion compensatedblock. In such a case, a motion compensation prediction unit (353) canaccess reference picture memory (357) to fetch samples used forprediction. After motion compensating the fetched samples in accordancewith the symbols (321) pertaining to the block, these samples can beadded by the aggregator (355) to the output of the scaler/inversetransform unit (351) (in this case called the residual samples orresidual signal) so as to generate output sample information. Theaddresses within the reference picture memory (357) from where themotion compensation prediction unit (353) fetches prediction samples canbe controlled by motion vectors, available to the motion compensationprediction unit (353) in the form of symbols (321) that can have, forexample X, Y, and reference picture components. Motion compensation alsocan include interpolation of sample values as fetched from the referencepicture memory (357) when sub-sample exact motion vectors are in use,motion vector prediction mechanisms, and so forth.

The output samples of the aggregator (355) can be subject to variousloop filtering techniques in the loop filter unit (356). Videocompression technologies can include in-loop filter technologies thatare controlled by parameters included in the coded video sequence (alsoreferred to as coded video bitstream) and made available to the loopfilter unit (356) as symbols (321) from the parser (320), but can alsobe responsive to meta-information obtained during the decoding ofprevious (in decoding order) parts of the coded picture or coded videosequence, as well as responsive to previously reconstructed andloop-filtered sample values.

The output of the loop filter unit (356) can be a sample stream that canbe output to the render device (312) as well as stored in the referencepicture memory (357) for use in future inter-picture prediction.

Certain coded pictures, once fully reconstructed, can be used asreference pictures for future prediction. For example, once a codedpicture corresponding to a current picture is fully reconstructed andthe coded picture has been identified as a reference picture (by, forexample, the parser (320)), the current picture buffer (358) can becomea part of the reference picture memory (357), and a fresh currentpicture buffer can be reallocated before commencing the reconstructionof the following coded picture.

The video decoder (310) may perform decoding operations according to apredetermined video compression technology in a standard, such as ITU-TRec. H.265. The coded video sequence may conform to a syntax specifiedby the video compression technology or standard being used, in the sensethat the coded video sequence adheres to both the syntax of the videocompression technology or standard and the profiles as document in thevideo compression technology or standard. Specifically, a profile canselect a certain tools as the only tools available for use under thatprofile from all the tools available in the video compression technologyor standard. Also necessary for compliance can be that the complexity ofthe coded video sequence is within bounds as defined by the level of thevideo compression technology or standard. In some cases, levels restrictthe maximum picture size, maximum frame rate, maximum reconstructionsample rate (measured in, for example megasamples per second), maximumreference picture size, and so on. Limits set by levels can, in somecases, be further restricted through Hypothetical Reference Decoder(HRD) specifications and metadata for HRD buffer management signaled inthe coded video sequence.

In an embodiment, the receiver (331) may receive additional (redundant)data with the encoded video. The additional data may be included as partof the coded video sequence(s). The additional data may be used by thevideo decoder (310) to properly decode the data and/or to moreaccurately reconstruct the original video data. Additional data can bein the form of, for example, temporal, spatial, or signal noise ratio(SNR) enhancement layers, redundant slices, redundant pictures, forwarderror correction codes, and so on.

FIG. 4 shows a block diagram of a video encoder (403) according to anembodiment of the present disclosure. The video encoder (403) isincluded in an electronic device (420). The electronic device (420)includes a transmitter (440) (e.g., transmitting circuitry). The videoencoder (403) can be used in the place of the video encoder (203) in theFIG. 2 example.

The video encoder (403) may receive video samples from a video source(401)(that is not part of the electronic device (420) in the FIG. 4example) that may capture video image(s) to be coded by the videoencoder (403). In another example, the video source (401) is a part ofthe electronic device (420).

The video source (401) may provide the source video sequence to be codedby the video encoder (403) in the form of a digital video sample streamthat can be of any suitable bit depth (for example: 8 bit, 10 bit, 12bit, . . . ), any colorspace (for example, BT.601 Y CrCB, RGB, . . . )and any suitable sampling structure (for example Y CrCb 4:2:0, Y CrCb4:4:4). In a media serving system, the video source (401) may be astorage device storing previously prepared video. In a videoconferencingsystem, the video source (401) may be a camera that captures local imageinformation as a video sequence. Video data may be provided as aplurality of individual pictures that impart motion when viewed insequence. The pictures themselves may be organized as a spatial array ofpixels, wherein each pixel can comprise one or more samples depending onthe sampling structure, color space, etc. in use. A person skilled inthe art can readily understand the relationship between pixels andsamples. The description below focusses on samples.

According to an embodiment, the video encoder (403) may code andcompress the pictures of the source video sequence into a coded videosequence (443) in real time or under any other time constraints asrequired by the application. Enforcing appropriate coding speed is onefunction of a controller (450). In some embodiments, the controller(450) controls other functional units as described below and isfunctionally coupled to the other functional units. The coupling is notdepicted for clarity. Parameters set by the controller (450) can includerate control related parameters (picture skip, quantizer, lambda valueof rate-distortion optimization techniques, . . . ), picture size, groupof pictures (GOP) layout, maximum motion vector search range, and soforth. The controller (450) can be configured to have other suitablefunctions that pertain to the video encoder (403) optimized for acertain system design.

In some embodiments, the video encoder (403) is configured to operate ina coding loop. As an oversimplified description, in an example, thecoding loop can include a source coder (430) (e.g., responsible forcreating symbols, such as a symbol stream, based on an input picture tobe coded, and a reference picture(s)), and a (local) decoder (433)embedded in the video encoder (403). The decoder (433) reconstructs thesymbols to create the sample data in a similar manner as a (remote)decoder also would create (as any compression between symbols and codedvideo bitstream is lossless in the video compression technologiesconsidered in the disclosed subject matter). The reconstructed samplestream (sample data) is input to the reference picture memory (434). Asthe decoding of a symbol stream leads to bit-exact results independentof decoder location (local or remote), the content in the referencepicture memory (434) is also bit exact between the local encoder andremote encoder. In other words, the prediction part of an encoder “sees”as reference picture samples exactly the same sample values as a decoderwould “see” when using prediction during decoding. This fundamentalprinciple of reference picture synchronicity (and resulting drift, ifsynchronicity cannot be maintained, for example because of channelerrors) is used in some related arts as well.

The operation of the “local” decoder (433) can be the same as of a“remote” decoder, such as the video decoder (310), which has alreadybeen described in detail above in conjunction with FIG. 3. Brieflyreferring also to FIG. 3, however, as symbols are available andencoding/decoding of symbols to a coded video sequence by an entropycoder (445) and the parser (320) can be lossless, the entropy decodingparts of the video decoder (310), including the buffer memory (315), andparser (320) may not be fully implemented in the local decoder (433).

An observation that can be made at this point is that any decodertechnology except the parsing/entropy decoding that is present in adecoder also necessarily needs to be present, in substantially identicalfunctional form, in a corresponding encoder. For this reason, thedisclosed subject matter focuses on decoder operation. The descriptionof encoder technologies can be abbreviated as they are the inverse ofthe comprehensively described decoder technologies. Only in certainareas a more detail description is required and provided below.

During operation, in some examples, the source coder (430) may performmotion compensated predictive coding, which codes an input picturepredictively with reference to one or more previously-coded picture fromthe video sequence that were designated as “reference pictures”. In thismanner, the coding engine (432) codes differences between pixel blocksof an input picture and pixel blocks of reference picture(s) that may beselected as prediction reference(s) to the input picture.

The local video decoder (433) may decode coded video data of picturesthat may be designated as reference pictures, based on symbols createdby the source coder (430). Operations of the coding engine (432) mayadvantageously be lossy processes. When the coded video data may bedecoded at a video decoder (not shown in FIG. 4), the reconstructedvideo sequence typically may be a replica of the source video sequencewith some errors. The local video decoder (433) replicates decodingprocesses that may be performed by the video decoder on referencepictures and may cause reconstructed reference pictures to be stored inthe reference picture cache (434). In this manner, the video encoder(403) may store copies of reconstructed reference pictures locally thathave common content as the reconstructed reference pictures that will beobtained by a far-end video decoder (absent transmission errors).

The predictor (435) may perform prediction searches for the codingengine (432). That is, for a new picture to be coded, the predictor(435) may search the reference picture memory (434) for sample data (ascandidate reference pixel blocks) or certain metadata such as referencepicture motion vectors, block shapes, and so on, that may serve as anappropriate prediction reference for the new pictures. The predictor(435) may operate on a sample block-by-pixel block basis to findappropriate prediction references. In some cases, as determined bysearch results obtained by the predictor (435), an input picture mayhave prediction references drawn from multiple reference pictures storedin the reference picture memory (434).

The controller (450) may manage coding operations of the source coder(430), including, for example, setting of parameters and subgroupparameters used for encoding the video data.

Output of all aforementioned functional units may be subjected toentropy coding in the entropy coder (445). The entropy coder (445)translates the symbols as generated by the various functional units intoa coded video sequence, by lossless compressing the symbols according totechnologies known to a person skilled in the art as, for exampleHuffman coding, variable length coding, arithmetic coding, and so forth.

The transmitter (440) may buffer the coded video sequence(s) as createdby the entropy coder (445) to prepare for transmission via acommunication channel (460), which may be a hardware/software link to astorage device which would store the encoded video data. The transmitter(440) may merge coded video data from the video coder (403) with otherdata to be transmitted, for example, coded audio data and/or ancillarydata streams (sources not shown).

The controller (450) may manage operation of the video encoder (403).During coding, the controller (450) may assign to each coded picture acertain coded picture type, which may affect the coding techniques thatmay be applied to the respective picture. For example, pictures oftenmay be assigned as one of the following picture types:

An Intra Picture (I picture) may be one that may be coded and decodedwithout using any other picture in the sequence as a source ofprediction. Some video codecs allow for different types of Intrapictures, including, for example Independent Decoder Refresh (“IDR”)Pictures. A person skilled in the art is aware of those variants of Ipictures and their respective applications and features.

A Predictive picture (P picture) may be one that may be coded anddecoded using intra prediction or inter prediction using at most onemotion vector and reference index to predict the sample values of eachblock.

A Bi-directionally Predictive Picture (B Picture) may be one that may becoded and decoded using intra prediction or inter prediction using atmost two motion vectors and reference indices to predict the samplevalues of each block. Similarly, multiple-predictive pictures can usemore than two reference pictures and associated metadata for thereconstruction of a single block.

Source pictures commonly may be subdivided spatially into a plurality ofsample blocks (for example, blocks of 4×4, 8×8, 4×8, or 16×16 sampleseach) and coded on a block-by-block basis. Blocks may be codedpredictively with reference to other (already coded) blocks asdetermined by the coding assignment applied to the blocks' respectivepictures. For example, blocks of I pictures may be codednon-predictively or they may be coded predictively with reference toalready coded blocks of the same picture (spatial prediction or intraprediction). Pixel blocks of P pictures may be coded predictively, viaspatial prediction or via temporal prediction with reference to onepreviously coded reference pictures. Blocks of B pictures may be codedpredictively, via spatial prediction or via temporal prediction withreference to one or two previously coded reference pictures.

The video encoder (403) may perform coding operations according to apredetermined video coding technology or standard, such as ITU-T Rec.H.265. In its operation, the video encoder (403) may perform variouscompression operations, including predictive coding operations thatexploit temporal and spatial redundancies in the input video sequence.The coded video data, therefore, may conform to a syntax specified bythe video coding technology or standard being used.

In an embodiment, the transmitter (440) may transmit additional datawith the encoded video. The source coder (430) may include such data aspart of the coded video sequence. Additional data may comprisetemporal/spatial/SNR enhancement layers, other forms of redundant datasuch as redundant pictures and slices, Supplementary EnhancementInformation (SEI) messages, Visual Usability Information (VUI) parameterset fragments, and so on.

A video may be captured as a plurality of source pictures (videopictures) in a temporal sequence. Intra-picture prediction (oftenabbreviated to Intra prediction) makes uses of spatial correlation in agiven picture, and inter-picture prediction makes uses of the (temporalor other) correlation between the pictures. In an example, a specificpicture under encoding/decoding, which is referred to as a currentpicture, is partitioned into blocks. When a block in the current pictureis similar to a reference block in a previously coded and still bufferedreference picture in the video, the block in the current picture can becoded by a vector that is referred to as a motion vector. The motionvector points to the reference block in the reference picture, and canhave a third dimension identifying the reference picture, in casemultiple reference pictures are in use.

In some embodiments, a bi-prediction technique can be used in theinter-picture prediction. According to the bi-prediction technique, tworeference pictures, such as a first and a second reference picture thatare both prior in decoding order to the current picture in the video(but may be in the past and future, respectively, in display order) areused. A block in the current picture can be coded by a first motionvector that points to a first reference block in the first referencepicture, and a second motion vector that points to a second referenceblock in the second reference picture. The block can be predicted by acombination of the first reference block and the second reference block.

Further, a merge mode technique can be used in the inter-pictureprediction to improve coding efficiency.

According to some embodiments of the disclosure, predictions, such asinter-picture predictions and intra-picture predictions are performed inthe unit of blocks. For example, according to the HEVC standard, apicture in a sequence of video pictures is partitioned into coding treeunits (CTU) for compression, the CTUs in a picture have the same size,such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTUincludes three coding tree blocks (CTBs), which are one luma CTB and twochroma CTBs. Each CTU can be recursively quadtree split into one ormultiple coding units (CUs). For example, a CTU of 64×64 pixels can besplit into one CU of 64×64 pixels, or 4 CUs of 32×32 pixels, or 16 CUsof 16×16 pixels. In an example, each CU is analyzed to determine aprediction type for the CU, such as an inter prediction type or an intraprediction type. The CU is split into one or more prediction units (PUs)depending on the temporal and/or spatial predictability. Generally, eachPU includes a luma prediction block (PB), and two chroma PBs. In anembodiment, a prediction operation in coding (encoding/decoding) isperformed in the unit of a prediction block. Using a luma predictionblock as an example of a prediction block, the prediction block includesa matrix of values (e.g., luma values) for pixels, such as 8×8 pixels,16×16 pixels, 8×16 pixels, 16×8 pixels and the like.

FIG. 5 shows a diagram of a video encoder (503) according to anotherembodiment of the disclosure. The video encoder (503) is configured toreceive a processing block (e.g., a prediction block) of sample valueswithin a current video picture in a sequence of video pictures, andencode the processing block into a coded picture that is part of a codedvideo sequence. In an example, the video encoder (503) is used in theplace of the video encoder (203) in the FIG. 2 example.

In an HEVC example, the video encoder (503) receives a matrix of samplevalues for a processing block, such as a prediction block of 8×8samples, and the like. The video encoder (503) determines whether theprocessing block is best coded using intra mode, inter mode, orbi-prediction mode using, for example, rate-distortion optimization.When the processing block is to be coded in intra mode, the videoencoder (503) may use an intra prediction technique to encode theprocessing block into the coded picture; and when the processing blockis to be coded in inter mode or bi-prediction mode, the video encoder(503) may use an inter prediction or bi-prediction technique,respectively, to encode the processing block into the coded picture. Incertain video coding technologies, merge mode can be an inter pictureprediction submode where the motion vector is derived from one or moremotion vector predictors without the benefit of a coded motion vectorcomponent outside the predictors. In certain other video codingtechnologies, a motion vector component applicable to the subject blockmay be present. In an example, the video encoder (503) includes othercomponents, such as a mode decision module (not shown) to determine themode of the processing blocks.

In the FIG. 5 example, the video encoder (503) includes the interencoder (530), an intra encoder (522), a residue calculator (523), aswitch (526), a residue encoder (524), a general controller (521) and anentropy encoder (525) coupled together as shown in FIG. 5.

The inter encoder (530) is configured to receive the samples of thecurrent block (e.g., a processing block), compare the block to one ormore reference blocks in reference pictures (e.g., blocks in previouspictures and later pictures), generate inter prediction information(e.g., description of redundant information according to inter encodingtechnique, motion vectors, merge mode information), and calculate interprediction results (e.g., predicted block) based on the inter predictioninformation using any suitable technique.

The intra encoder (522) is configured to receive the samples of thecurrent block (e.g., a processing block), in some cases compare theblock to blocks already coded in the same picture, generate quantizedcoefficients after transform and, in some cases also intra predictioninformation (e.g., an intra prediction direction information accordingto one or more intra encoding techniques).

The general controller (521) is configured to determine general controldata and control other components of the video encoder (503) based onthe general control data. In an example, the general controller (521)determines the mode of the block, and provides a control signal to theswitch (526) based on the mode. For example, when the mode is the intra,the general controller (521) controls the switch (526) to select theintra mode result for use by the residue calculator (523), and controlsthe entropy encoder (525) to select the intra prediction information andinclude the intra prediction information in the bitstream; and when themode is the inter mode, the general controller (521) controls the switch(526) to select the inter prediction result for use by the residuecalculator (523), and controls the entropy encoder (525) to select theinter prediction information and include the inter predictioninformation in the bitstream.

The residue calculator (523) is configured to calculate a difference(residue data) between the received block and prediction resultsselected from the intra encoder (522) or the inter encoder (530). Theresidue encoder (524) is configured to operate based on the residue datato encode the residue data to generate the transform coefficients. In anexample, the residue encoder (524) is configured to convert the residuedata in the frequency domain, and generate the transform coefficients.The transform coefficients are then subject to quantization processingto obtain quantized transform coefficients.

The entropy encoder (525) is configured to format the bitstream toinclude the encoded block. The entropy encoder (525) is configured toinclude various information according to a suitable standard, such asHEVC standard. In an example, the entropy encoder (525) is configured toinclude the general control data, the selected prediction information(e.g., intra prediction information or inter prediction information),the residue information, and other suitable information in thebitstream. Note that, according to the disclosed subject matter, whencoding a block in the merge submode of either inter mode orbi-prediction mode, there is no residue information.

FIG. 6 shows a diagram of a video decoder (610) according to anotherembodiment of the disclosure. The video decoder (610) is configured toreceive a coded pictures that are part of a coded video sequence, anddecode the coded picture to generate a reconstructed picture. In anexample, the video decoder (610) is used in the place of the videodecoder (210) in the FIG. 2 example.

In the FIG. 6 example, the video decoder (610) includes an entropydecoder (671), an inter decoder (680), a residue decoder (673), areconstruction module (674), and an intra decoder (672) coupled togetheras shown in FIG. 6.

The entropy decoder (671) can be configured to reconstruct, from thecoded picture, certain symbols that represent the syntax elements ofwhich the coded picture is made up. Such symbols can include, forexample, the mode in which a block is coded (such as, for example,intra, inter, b-predicted, the latter two in merge submode or anothersubmode), prediction information (such as, for example, intra predictioninformation or inter prediction information) that can identify certainsample or metadata that is used for prediction by the intra decoder(672) or the inter decoder (680) respectively residual information inthe form of, for example, quantized transform coefficients, and thelike. In an example, when the prediction mode is inter or bi-predictedmode, the inter prediction information is provided to the inter decoder(680); and when the prediction type is the intra prediction type, theintra prediction information is provided to the intra decoder (672). Theresidual information can be subject to inverse quantization and isprovided to the residue decoder (673).

The inter decoder (680) is configured to receive the inter predictioninformation, and generate inter prediction results based on the interprediction information.

The intra decoder (672) is configured to receive the intra predictioninformation, and generate prediction results based on the intraprediction information.

The residue decoder (673) is configured to perform inverse quantizationto extract de-quantized transform coefficients, and process thede-quantized transform coefficients to convert the residual from thefrequency domain to the spatial domain. The residue decoder (673) mayalso require certain control information (to include the QuantizerParameter QP), and that information may be provided by the entropydecoder (671) (datapath not depicted as this may be low volume controlinformation only).

The reconstruction module (674) is configured to combine, in the spatialdomain, the residual as output by the residue decoder (673) and theprediction results (as output by the inter or intra prediction modulesas the case may be) to form a reconstructed block, that may be part ofthe reconstructed picture, which in turn may be part of thereconstructed video. It is noted that other suitable operations, such asa deblocking operation and the like, can be performed to improve thevisual quality.

It is noted that the video encoders (203), (403) and (503), and thevideo decoders (210), (310) and (610) can be implemented using anysuitable technique. In an embodiment, the video encoders (203), (403)and (503), and the video decoders (210), (310) and (610) can beimplemented using one or more integrated circuits. In anotherembodiment, the video encoders (203), (403) and (403), and the videodecoders (210), (310) and (610) can be implemented using one or moreprocessors that execute software instructions.

Block based compensation from a different picture may be referred to asmotion compensation. Block compensation may also be done from apreviously reconstructed area within the same picture, which may bereferred to as intra picture block compensation or intra block copy. Forexample, a displacement vector that indicates an offset between acurrent block and the reference block is referred to as a block vector.According to some embodiments, a block vector points to a referenceblock that is already reconstructed and available for reference. Also,for parallel processing consideration, a reference area that is beyond atile/slice boundary or wavefront ladder-shaped boundary may also beexcluded from being referenced by the block vector. Due to theseconstraints, a block vector may be different from a motion vector (MV)in motion compensation, where the motion vector can be at any value(positive or negative, at either x or y direction).

FIG. 7 illustrates an embodiment of intra picture block compensation(e.g., intra block copy mode). In FIG. 7, a current picture 700 includesa set of blocks that have already been coded/decoded (i.e., gray coloredblocks) and a set of blocks that have yet to be coded/decoded (i.e.,white colored blocks). A sub-block 702 of one of the blocks that haveyet to be coded/decoded may be associated with a block vector 704 thatpoints to another sub-block 706 that has previously been coded/decoded.Accordingly, any motion information associated with the sub-block 706may be used for the coding/decoding of sub-block 702.

According to some embodiments, the coding of a block vector is explicit.In other embodiments, the coding of the block vector is implicit. In theexplicit mode, the difference between a block vector and its predictoris signaled, whereas in the implicit mode, the block vector is recoveredfrom its predictor in a similar way as a motion vector prediction inmerge mode. The resolution of a block vector, in some embodiments, isrestricted to integer positions. In other embodiments, the block vectorpoints to fractional positions.

According to some embodiments, the use of the intra picture blockcompensation (i.e., intra block copy mode) at the block level, issignaled using a reference index, where a current decoded picture istreated as a reference picture, which is put in a last position of areference picture list. This reference picture may also be managedtogether with other temporal reference pictures in a decoded picturebuffer (DPB).

According to some embodiments, a reference block is flipped horizontallyor vertically before being used to predict a current block (e.g.,flipped intra block copy). In some embodiments, each compensation unitinside an M×N coding block is an M×1 or 1×N line (e.g., line based intrablock copy).

According to some embodiments, motion compensation is performed at theblock level, where the current block is the processing unit forperforming motion compensation using the same motion information. Inthis regard, given the size of a block, all pixels in the block will usethe same motion information to form their prediction block. Examples ofblock level motion compensation include using spatial merge candidates,temporal candidates, and in bi-directional prediction, combinations ofmotion vectors from existing merge candidates.

Referring to FIG. 8, a current block (801) comprises samples that havebeen found by the encoder/decoder during the motion search process to bepredictable from a previous block of the same size that has beenspatially shifted. In some embodiments, instead of coding that MVdirectly, the MV can be directed from metadata associated with one ormore reference pictures, for example, from a most recent (in decodingorder) reference picture, using the MV associated with either one offive surrounding samples, denoted A0, A1, and B0, B1, B2 (802 through806, respectively). The blocks A0, A1, B0, B1, and B2 may be referred toas spatial merge candidates.

According to some embodiments, pixels (e.g., sub-blocks) at differentpositions inside a motion compensation block can have different motioninformation. These differences from the block level motion informationmay be derived instead of signaled. This type of motion compensation maybe referred to as sub-block level motion compensation, which allows formotion compensation of a block to be smaller than the block itself. Inthis regard, each block may have multiple sub-blocks, each of which maycontain different motion information.

An example of sub-block level motion compensation includes sub-blockbased temporal motion vector prediction, where sub-blocks of a currentblock have different motion vectors. Another example of sub-block levelmotion compensation is ATMVP, which is a method that allows each codingblock to fetch multiple sets of motion information from multiple blockssmaller than the current coding block from a collocated referencepicture.

Another example of sub-block level motion compensation includesspatial/temporal fusion with sub-block adjustment, where for eachsub-block in a current block, its motion vector is adjusted according toits spatial/temporal neighbor's motion vectors. In this mode, for somesub-blocks, the motion information from a corresponding sub-block in atemporal reference picture may be needed.

Another example of sub-block level motion compensation is affine codedmotion compensation block, where according to the neighboring blocks'motion vectors, the motion vectors at the four corners of a currentblock are derived first. Subsequently, the rest of the current block'smotion vectors (e.g., at sub-block or pixel level) are derived using anaffine model such that each sub-block can have a different motion vectoras compared to the neighbors of a respective sub-block.

Another example of sub-block level motion compensation is mergecandidate refinement using decoder side motion vector derivation. Inthis mode, after getting the motion vector predictor(s) for the currentblock or the sub-blocks of the current block, methods such as templatematching or bilateral matching can be used to further refine the givenmotion vector predictor(s). The refined motion vector(s) may be used toperform motion compensation. The same refinement operation may beperformed at both encoder and decoder sides so that no additionalinformation is needed by the decoder regarding how the refinement isdisplaced from the original predictor. Furthermore, the skip mode may beconsidered as a special merge mode, where in addition to deriving motioninformation of a current block from the current block's neighbors, theprediction residue of the current block is also zero.

According to some embodiments, in sub-block temporal motion vectorprediction, sub-blocks of a current block may have different motionvector predictors, which are derived from a temporal reference picture.For example, a set of motion information, including a motion vector andan associated reference index for the current block is identified. Themotion information may be determined from a first available spatialmerge candidate. Using this motion information, a reference block in areference picture is determined for a current block. The reference blockis also divided into sub-blocks. In some embodiments, for each currentblock sub-block (CBSB) in the current picture, there is a correspondingreference block sub-block (RBSB) in the reference picture.

In some embodiments, for each CBSB, if the corresponding RBSB is codedin an inter mode with a set of motion information, then that motioninformation is converted (e.g., using methods such as motion vectorscaling in temporal motion vector prediction, etc.) and used as apredictor for the motion vector of this CBSB. The method for handling aRBSB that is coded in an intra mode (e.g., intra block copy mode) isdescribed in further detail below.

According to some embodiments, when sub-block based temporal motionvector prediction mode is used, each CBSB is not permitted to be codedin an intra mode such as the intra block copy mode. This may beaccomplished by treating an intra block copy coded RBSB as intra mode.Particularly, regardless of how the intra block copy mode is considered(e.g., as inter mode, intra mode, or a third mode), for the CBSB, whenits corresponding RBSB is coded in the intra block copy mode, this RBSBis considered as intra mode in the sub-block based temporal motionvector prediction. Therefore, in some embodiments, the RBSB that iscoded in intra block copy mode is handled in accordance with a defaultsetting in the sub-block based temporal motion vector prediction. Forexample, when a corresponding RBSB of a CBSB is coded in the intra blockcopy mode, a default motion vector, such as the zero motion vector, maybe used as a predictor for the CBSB. In this example, the referencepicture for the CBSB will no longer be the current picture, but atemporal reference picture. For example, the temporal reference picturemay be a picture that is shared by all sub-blocks of the current block,the first reference picture in the reference picture list, theco-located picture for TMVP purpose, etc. In another example, for theCBSB, when its corresponding RBSB is coded in the inter mode, but thereference picture is the current picture, a default motion vector, suchas the zero motion vector, is assigned to the CBSB. In this regard, eventhough the RBSB is coded in the inter mode, since the current picture isthe same as the reference picture, the RBSB is processed as if the RBSBwere coded in the intra mode.

FIG. 9 illustrates an example of performing sub-block based temporalmotion vector prediction. FIG. 9 illustrates a current picture 900 thathas nine blocks including a current block 900A. The current block 900Ais divided into four sub-blocks 1-4. The current picture 900 may beassociated with reference picture 902, which includes nine previouslycoded/decoded blocks. Furthermore, as illustrated in FIG. 9, the currentblock 900A has a motion vector 904 that points to reference block 902A.The motion vector 904 may be determined from using the motion vectors ofone or more neighboring blocks of the current block 900A (e.g., spatialmerge candidates). The reference block 902A is divided into foursub-blocks 1-4. The sub-blocks 1-4 of reference block 902A correspond tothe sub-blocks 1-4 of current block 900A, respectively. If the referencepicture 902 is the same as the current picture 900, each RBSB in block902A is treated as if these blocks were coded in the intra mode. In thisregard, for example, if sub-block 1 of block 902A is coded in the intermode, but the reference picture 902 is the same as the current picture900, sub-block 1 of block 902A is treated as if this sub-block werecoded in intra mode where a default motion vector is assigned tosub-block 1 of block 900A.

If the reference picture 902 is different from the current picture 900,the sub-blocks 1-4 of reference block 902A may be used for performingsub-block based temporal motion vector prediction for sub-blocks 1-4 ofblock 900A, respectively. For example, the motion vector predictor ofsub-block 1 of current block 900A is determined based on whethersub-block 1 of reference block 902A is coded in an inter mode or anintra mode (e.g., intra block copy mode). If sub-block 1 of referenceblock 902A is coded with the inter mode, the motion vector of sub-block1 of reference block 902A is used to determine the motion vector ofsub-block 1 of current block 900A. However, if sub-block 1 of referenceblock 902A is coded with the intra mode, the motion vector of sub-block1 of current block 900A is set to the zero motion vector.

FIG. 10 illustrates an embodiment of a process that may be performed byan encoder or decoder such as intra encoder 522 or intra decoder 672,respectively. The process may start at step S1000 where a currentpicture is acquired from a coded video bitstream. For example, referringto FIG. 9, the current picture 900 may be acquired from a coded videobit stream. The process proceeds to step S1002 where, for a currentblock in the current picture, a reference block from a reference pictureis identified. For example, referring to FIG. 9, the reference picture902 may be retrieved from a reference picture list associated withcurrent block 900A. When performing sub-block temporal motion vectorprediction on current block 900A, a motion vector 904 may be used toidentify the reference block 902A of reference picture 902.

The process proceeds to step S1004 where it is determined whether thereference picture is the same as the current picture. If the referencepicture is not the same as the current picture, the process proceeds tostep S1006 where a coding mode of a RBSB that corresponds to a CBSB isdetermined. For example, referring to FIG. 9, the coding mode of RBSB 1of the reference block 902A, which corresponds to CBSB 1 of currentblock 900A, is determined. The process proceeds to step S1008 where itis determined whether the coding mode of the RBSB is the inter mode. Ifthe coding mode of the RBSB is the inter mode, the process proceeds tostep S1010 where a motion vector predictor for of the CBSB is determinedbased on a motion vector predictor of the RBSB. For example, if thecoding mode of RBSB 1 of the reference block 902A is the inter mode, themotion vector predictor of CBSB 1 of the current block 900A isdetermined based on the motion vector predictor or RBSB 1 of thereference block 902A. For example, the motion vector predictor of RBSB 1of the reference block 902A is converted (e.g., using methods such asmotion vector scaling in temporal motion vector prediction, etc.) andused as a motion vector predictor for the CBSB 1 of current block 900A.

Returning to step S1008, if the coding mode of the RBSB is not the intermode (e.g., coding mode of RBSB is the intra mode), the process proceedsto step S1012 where the motion vector predictor of the CBSB is set to adefault motion vector. For example, if RBSB 1 of the reference block902A is coded in the intra mode, the motion vector predictor of CBSB 1of the current block 900A is set to a default motion vector such as thezero motion vector.

Returning to step S1004, if the reference picture is the same as thecurrent picture, the process proceeds to step S1012 where the motionvector predictor of the CBSB is set to a default motion vector. In thisregard, when the reference picture is the same as the current picture,the coding mode of the RBSB is determined to be as the intra mode wherethe motion vector predictor for the corresponding CBSB is set to thedefault motion vector. In this regard, even if the RBSB is in the intermode, the RBSB is treated as if the RBSB were coded in the intra mode bysetting the motion vector predictor of the CBSB to a default motionvector. The steps S1004 to S1012 may be repeated for each sub-block inthe current block 900A.

The techniques described above, can be implemented as computer softwareusing computer-readable instructions and physically stored in one ormore computer-readable media. For example, FIG. 11 shows a computersystem (1100) suitable for implementing certain embodiments of thedisclosed subject matter.

The computer software can be coded using any suitable machine code orcomputer language, that may be subject to assembly, compilation,linking, or like mechanisms to create code comprising instructions thatcan be executed directly, or through interpretation, micro-codeexecution, and the like, by one or more computer central processingunits (CPUs), Graphics Processing Units (GPUs), and the like.

The instructions can be executed on various types of computers orcomponents thereof, including, for example, personal computers, tabletcomputers, servers, smartphones, gaming devices, internet of thingsdevices, and the like.

The components shown in FIG. 11 for computer system (1100) are exemplaryin nature and are not intended to suggest any limitation as to the scopeof use or functionality of the computer software implementingembodiments of the present disclosure. Neither should the configurationof components be interpreted as having any dependency or requirementrelating to any one or combination of components illustrated in theexemplary embodiment of a computer system (1100).

Computer system (1100) may include certain human interface inputdevices. Such a human interface input device may be responsive to inputby one or more human users through, for example, tactile input (such as:keystrokes, swipes, data glove movements), audio input (such as: voice,clapping), visual input (such as: gestures), olfactory input (notdepicted). The human interface devices can also be used to capturecertain media not necessarily directly related to conscious input by ahuman, such as audio (such as: speech, music, ambient sound), images(such as: scanned images, photographic images obtain from a still imagecamera), video (such as two-dimensional video, three-dimensional videoincluding stereoscopic video).

Input human interface devices may include one or more of (only one ofeach depicted): keyboard (1101), mouse (1102), trackpad (1103), touchscreen (1110), data-glove (not shown), joystick (1105), microphone(1106), scanner (1107), camera (1108).

Computer system (1100) may also include certain human interface outputdevices. Such human interface output devices may be stimulating thesenses of one or more human users through, for example, tactile output,sound, light, and smell/taste. Such human interface output devices mayinclude tactile output devices (for example tactile feedback by thetouch-screen (1110), data-glove (not shown), or joystick (1105), butthere can also be tactile feedback devices that do not serve as inputdevices), audio output devices (such as: speakers (1109), headphones(not depicted)), visual output devices (such as screens (1110) toinclude CRT screens, LCD screens, plasma screens, OLED screens, eachwith or without touch-screen input capability, each with or withouttactile feedback capability—some of which may be capable to output twodimensional visual output or more than three dimensional output throughmeans such as stereographic output; virtual-reality glasses (notdepicted), holographic displays and smoke tanks (not depicted)), andprinters (not depicted).

Computer system (1100) can also include human accessible storage devicesand their associated media such as optical media including CD/DVD ROM/RW(1120) with CD/DVD or the like media (1121), thumb-drive (1122),removable hard drive or solid state drive (1123), legacy magnetic mediasuch as tape and floppy disc (not depicted), specialized ROM/ASIC/PLDbased devices such as security dongles (not depicted), and the like.

Those skilled in the art should also understand that term “computerreadable media” as used in connection with the presently disclosedsubject matter does not encompass transmission media, carrier waves, orother transitory signals.

Computer system (1100) can also include an interface to one or morecommunication networks. Networks can for example be wireless, wireline,optical. Networks can further be local, wide-area, metropolitan,vehicular and industrial, real-time, delay-tolerant, and so on. Examplesof networks include local area networks such as Ethernet, wireless LANs,cellular networks to include GSM, 3G, 4G, 5G, LTE and the like, TVwireline or wireless wide area digital networks to include cable TV,satellite TV, and terrestrial broadcast TV, vehicular and industrial toinclude CANBus, and so forth. Certain networks commonly require externalnetwork interface adapters that attached to certain general purpose dataports or peripheral buses (1149) (such as, for example USB ports of thecomputer system (1100)); others are commonly integrated into the core ofthe computer system (1100) by attachment to a system bus as describedbelow (for example Ethernet interface into a PC computer system orcellular network interface into a smartphone computer system). Using anyof these networks, computer system (1100) can communicate with otherentities. Such communication can be uni-directional, receive only (forexample, broadcast TV), uni-directional send-only (for example CANbus tocertain CANbus devices), or bi-directional, for example to othercomputer systems using local or wide area digital networks. Certainprotocols and protocol stacks can be used on each of those networks andnetwork interfaces as described above.

Aforementioned human interface devices, human-accessible storagedevices, and network interfaces can be attached to a core (1140) of thecomputer system (1100).

The core (1140) can include one or more Central Processing Units (CPU)(1141), Graphics Processing Units (GPU) (1142), specialized programmableprocessing units in the form of Field Programmable Gate Areas (FPGA)(1143), hardware accelerators for certain tasks (1144), and so forth.These devices, along with Read-only memory (ROM) (1145), Random-accessmemory (1146), internal mass storage such as internal non-useraccessible hard drives, SSDs, and the like (1147), may be connectedthrough a system bus (1148). In some computer systems, the system bus(1148) can be accessible in the form of one or more physical plugs toenable extensions by additional CPUs, GPU, and the like. The peripheraldevices can be attached either directly to the core's system bus (1148),or through a peripheral bus (1149). Architectures for a peripheral businclude PCI, USB, and the like.

CPUs (1141), GPUs (1142), FPGAs (1143), and accelerators (1144) canexecute certain instructions that, in combination, can make up theaforementioned computer code. That computer code can be stored in ROM(1145) or RAM (1146). Transitional data can be also be stored in RAM(1146), whereas permanent data can be stored for example, in theinternal mass storage (1147). Fast storage and retrieve to any of thememory devices can be enabled through the use of cache memory, that canbe closely associated with one or more CPU (1141), GPU (1142), massstorage (1147), ROM (1145), RAM (1146), and the like.

The computer readable media can have computer code thereon forperforming various computer-implemented operations. The media andcomputer code can be those specially designed and constructed for thepurposes of the present disclosure, or they can be of the kind wellknown and available to those having skill in the computer software arts.

As an example and not by way of limitation, the computer system havingarchitecture (1100), and specifically the core (1140) can providefunctionality as a result of processor(s) (including CPUs, GPUs, FPGA,accelerators, and the like) executing software embodied in one or moretangible, computer-readable media. Such computer-readable media can bemedia associated with user-accessible mass storage as introduced above,as well as certain storage of the core (1140) that are of non-transitorynature, such as core-internal mass storage (1147) or ROM (1145). Thesoftware implementing various embodiments of the present disclosure canbe stored in such devices and executed by core (1140). Acomputer-readable medium can include one or more memory devices orchips, according to particular needs. The software can cause the core(1140) and specifically the processors therein (including CPU, GPU,FPGA, and the like) to execute particular processes or particular partsof particular processes described herein, including defining datastructures stored in RAM (1146) and modifying such data structuresaccording to the processes defined by the software. In addition or as analternative, the computer system can provide functionality as a resultof logic hardwired or otherwise embodied in a circuit (for example:accelerator (1144)), which can operate in place of or together withsoftware to execute particular processes or particular parts ofparticular processes described herein. Reference to software canencompass logic, and vice versa, where appropriate. Reference to acomputer-readable media can encompass a circuit (such as an integratedcircuit (IC)) storing software for execution, a circuit embodying logicfor execution, or both, where appropriate. The present disclosureencompasses any suitable combination of hardware and software.

APPENDIX A: ACRONYMS MV: Motion Vector HEVC: High Efficiency VideoCoding SEI: Supplementary Enhancement Information VUI: Video UsabilityInformation GOPs: Groups of Pictures TUs: Transform Units, PUs:Prediction Units CTUs: Coding Tree Units CTBs: Coding Tree Blocks PBs:Prediction Blocks HRD: Hypothetical Reference Decoder SNR: Signal NoiseRatio CPUs: Central Processing Units GPUs: Graphics Processing UnitsCRT: Cathode Ray Tube LCD: Liquid-Crystal Display OLED: OrganicLight-Emitting Diode CD: Compact Disc DVD: Digital Video Disc ROM:Read-Only Memory RAM: Random Access Memory ASIC: Application-SpecificIntegrated Circuit PLD: Programmable Logic Device LAN: Local AreaNetwork

GSM: Global System for Mobile communications

LTE: Long-Term Evolution CANBus: Controller Area Network Bus USB:Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: FieldProgrammable Gate Areas

SSD: solid-state drive

IC: Integrated Circuit CU: Coding Unit

While this disclosure has described several exemplary embodiments, thereare alterations, permutations, and various substitute equivalents, whichfall within the scope of the disclosure. It will thus be appreciatedthat those skilled in the art will be able to devise numerous systemsand methods which, although not explicitly shown or described herein,embody the principles of the disclosure and are thus within the spiritand scope thereof.

(1) A method of video decoding for a decoder includes acquiring acurrent picture from a coded video bitstream; identifying, for a currentblock included in the current picture, a reference block included in areference picture that is different from the current picture, thecurrent block being divided into a plurality of sub-blocks (CBSBs), thereference block having a plurality of sub-blocks (RBSBs) that eachcorrespond to a different one of the plurality of CBSBs; determiningwhether the reference picture for the RBSB is the current picture; inresponse to determining that the reference picture for the RBSB is thecurrent picture, determining a coding mode of the RBSB as an intra mode;and in response to determining that the reference picture for the RBSBis not the current picture, (i) determining, for one of the CBSBs,whether a coding mode of the RBSB is one of an intra mode and an intermode, and (ii) determining a motion vector predictor for the one of theCBSBs based on whether the coding mode of the corresponding RBSB is oneof the intra mode and the inter mode.

(2) The method of feature (1), in which in response to determining thatthe coding mode of the corresponding RBSB is the intra mode, thedetermined motion vector predictor for the one CBSB is set to a defaultmotion vector.

(3) The method according to feature (2), in which the default motionvector is one of (i) a zero motion vector and (ii) an offset between theCBSB and the RBSB.

(4) The method of any one of features (1)-(3), in which in response todetermining that the coding mode of the corresponding RBSB is the intermode, the determined motion vector predictor for the one CBSB is basedon a motion vector predictor associated with the corresponding RB SB.

(5) The method of feature (4), in which the determined motion vectorpredictor is a scaled version of the motion vector predictor associatedwith the corresponding RBSB.

(6) The method of any one of features (1)-(5), in which the referenceblock is identified in accordance with a motion vector predictorassociated with a block adjacent to the current block.

(7) The method of any one of features (1)-(6), in which the referencepicture is a first reference picture from a sequence of referencepictures associated with the current picture.

(8) A video decoder for video decoding, including processing circuitryconfigured to: acquire a current picture from a coded video bitstream,identify, for a current block included in the current picture, areference block included in a reference picture that is different fromthe current picture, the current block being divided into a plurality ofsub-blocks (CBSBs), the reference block having a plurality of sub-blocks(RBSBs) that each correspond to a different one of the plurality ofCBSBs, determine whether the reference picture for the RBSB is thecurrent picture, in response to the determination that the referencepicture for the RBSB is the current picture, determine a coding mode ofthe RBSB as an intra mode, and in response to the determination that thereference picture for the RBSB is not the current picture, (i)determine, for one of the CBSBs, whether a coding mode of the RBSB isone of an intra mode and an inter mode, and (ii) determine a motionvector predictor for the one of the CBSBs based on whether the codingmode of the corresponding RBSB is one of the intra mode and the intermode.

(9) The video decoder of feature (8), in which in response to thedetermination that the coding mode of the corresponding RBSB is theintra mode, the determined motion vector predictor for the one CBSB isset to a default motion vector.

(10) The video decoder according to feature (9), in which the defaultmotion vector is one of (i) a zero motion vector and (ii) an offsetbetween the CBSB and the RBSB.

(11) The video decoder of any one of features (8)-(10), in which inresponse to the determination that the coding mode of the correspondingRBSB is the inter mode, the determined motion vector predictor for theone CBSB is based on a motion vector predictor associated with thecorresponding RBSB.

(12) The video decoder of feature (11), in which the determined motionvector predictor is a scaled version of the motion vector predictorassociated with the corresponding RBSB.

(13) The video decoder of any one of features (8)-(12), in which thereference block is identified in accordance with a motion vectorpredictor associated with a block adjacent to the current block.

(14) The video decoder of any one of features (8)-(13), in which thereference picture is a first reference picture from a sequence ofreference pictures associated with the current picture.

(15) A non-transitory computer readable medium having instructionsstored therein, which when executed by a processor in a video decodercauses the processor to execute a method including acquiring a currentpicture from a coded video bitstream; identifying, for a current blockincluded in the current picture, a reference block included in areference picture that is different from the current picture, thecurrent block being divided into a plurality of sub-blocks (CBSBs), thereference block having a plurality of sub-blocks (RBSBs) that eachcorrespond to a different one of the plurality of CBSBs; determiningwhether the reference picture for the RBSB is the current picture; inresponse to determining that the reference picture for the RBSB is thecurrent picture, determining a coding mode of the RBSB as an intra mode;and in response to determining that the reference picture for the RBSBis not the current picture, (i) determining, for one of the CBSBs,whether a coding mode of the RBSB is one of an intra mode and an intermode, and (ii) determining a motion vector predictor for the one of theCBSBs based on whether the coding mode of the corresponding RBSB is oneof the intra mode and the inter mode.

(16) The non-transitory computer readable medium of feature (15), inwhich in response to determining that the coding mode of thecorresponding RBSB is the intra mode, the determined motion vectorpredictor for the one CBSB is set to a default motion vector.

(17) The non-transitory computer readable medium according to feature(16), in which the default motion vector is one of (i) a zero motionvector and (ii) an offset between the CBSB and the RBSB.

(18) The non-transitory computer readable medium of any one of features(15)-(17), in which in response to determining that the coding mode ofthe corresponding RBSB is the inter mode, the determined motion vectorpredictor for the one CBSB is based on a motion vector predictorassociated with the corresponding RBSB.

(19) The non-transitory computer readable medium of feature (18), inwhich the determined motion vector predictor is a scaled version of themotion vector predictor associated with the corresponding RBSB.

(20) The non-transitory computer readable medium of any one of features(15)-(19), in which the reference block is identified in accordance witha motion vector predictor associated with a block adjacent to thecurrent block.

1. (canceled)
 2. A method of video decoding for a decoder, the methodcomprising: acquiring a current picture from a coded video bitstream;identifying, for a current block included in the current picture, areference block included in a reference picture, the reference picturebeing different from the current picture, the current block beingdivided into a first plurality of sub-blocks (CBSBs) that are coded inan inter mode, the reference block having a second plurality ofsub-blocks (RBSBs), each CBSB corresponding to at least one RBSB;determining whether a corresponding RBSB is coded in an intra predictionmode or an intra block copy mode; in response to the corresponding RBSBbeing coded in the intra prediction mode or the intra block copy mode,setting a first motion vector predictor of the corresponding CBSB to azero motion vector; and decoding the plurality of CBSBs by performingsub-block based temporal motion vector prediction based on the firstmotion vector predictor for each CBSB.
 3. The method of claim 2, inresponse to the corresponding RBSB being coded in the inter mode,setting the first motion vector predictor of the corresponding CBSBusing a collocated second motion vector predictor of the correspondingRBSB.
 4. The method of claim 3, wherein the collocated second motionvector predictor of the corresponding RBSB is a scaled version of thecollocated second motion vector predictor of the corresponding RBSB. 5.The method of claim 2, wherein the reference block is identified inaccordance with a motion vector predictor associated with a blockadjacent to the current block.
 6. The method of claim 2, wherein thereference picture is a temporal picture.
 7. The method of claim 6,wherein the temporal picture is a first reference picture from asequence of reference pictures associated with the current picture. 8.The method of claim 6, wherein the temporal picture is a collatedpicture.
 9. The method of claim 6, wherein the temporal picture is apicture that is shared by all sub-blocks of the current block.
 10. Avideo decoder for video decoding, comprising: processing circuitryconfigured to: acquire a current picture from a coded video bitstream;identify, for a current block included in the current picture, areference block included in a reference picture, the reference picturebeing different from the current picture, the current block beingdivided into a first plurality of sub-blocks (CBSBs) that are coded inan inter mode, the reference block having a second plurality ofsub-blocks (RBSBs), each CBSB corresponding to at least one RBSB;determine whether a corresponding RBSB is coded in an intra predictionmode or an intra block copy mode; in response to the corresponding RBSBbeing coded in the intra prediction mode or the intra block copy mode,set a first motion vector predictor of the corresponding CBSB to a zeromotion vector; and decode the plurality of CBSBs by performing sub-blockbased temporal motion vector prediction based on the first motion vectorpredictor for each CBSB.
 11. The video decoder of claim 10, wherein theprocessing circuitry is further configured to, in response to thecorresponding RBSB being coded in the inter mode, set the first motionvector predictor of the corresponding CBSB using a collocated secondmotion vector predictor of the corresponding RBSB.
 12. The video decoderof claim 11, wherein the collocated second motion vector predictor ofthe corresponding RBSB is a scaled version of the collocated secondmotion vector predictor of the corresponding RBSB.
 13. The video decoderof claim 10, wherein the reference block is identified in accordancewith a motion vector predictor associated with a block adjacent to thecurrent block.
 14. The video decoder of claim 10, wherein the referencepicture is a temporal picture.
 15. The video decoder of claim 14,wherein the temporal picture is a first reference picture from asequence of reference pictures associated with the current picture. 16.The video decoder of claim 14, wherein the temporal picture is acollated picture.
 17. The video decoder of claim 14, wherein thetemporal picture is a picture that is shared by all sub-blocks of thecurrent block.
 18. A non-transitory computer readable medium havinginstructions stored therein, which when executed by a processor in avideo decoder causes the processor to execute a method of video decodingcomprising: acquiring a current picture from a coded video bitstream;identifying, for a current block included in the current picture, areference block included in a reference picture, the reference picturebeing different from the current picture, the current block beingdivided into a first plurality of sub-blocks (CBSBs) that are coded inan inter mode, the reference block having a second plurality ofsub-blocks (RBSBs), each CBSB corresponding to at least one RBSB;determining whether a corresponding RBSB is coded in an intra predictionmode or an intra block copy mode; in response to the corresponding RBSBbeing coded in the intra prediction mode or the intra block copy mode,setting a first motion vector predictor of the corresponding CBSB to azero motion vector; and decoding the plurality of CBSBs by performingsub-block based temporal motion vector prediction based on the firstmotion vector predictor for each CBSB.
 19. The non-transitory computerreadable medium of claim 18, in response to the corresponding RBSB beingcoded in the inter mode, setting the first motion vector predictor ofthe corresponding CBSB using a collocated second motion vector predictorof the corresponding RBSB.
 20. The non-transitory computer readablemedium of claim 19, wherein the collocated second motion vectorpredictor of the corresponding RBSB is a scaled version of the secondmotion vector predictor of the corresponding RBSB.
 21. Thenon-transitory computer readable medium of claim 18, wherein thereference block is identified in accordance with a motion vectorpredictor associated with a block adjacent to the current block.